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ABSTRACT 


Selecting courses in an open-curriculum education program 
is a difficult task for students and academic advisors. Course 
recommendation systems nowadays can be used to reduce 
the complexity of this task. To control the recommendation 
error, we argue that course recommendations need to be pro- 
vided together with statistical confidence. The latter can be 
used for computing a statistically valid set of recommended 
courses that contains courses a student is likely to take with 
a probability of at least 1—€ for a user-specified significance 
level «. For that purpose, we introduce a generic algorithm 
for course recommendation based on the conformal predic- 
tion framework. The algorithm is used for implementing two 
conformal course recommender systems. Through experi- 
mentation, we show that these systems accurately suggest 
courses to students while maintaining statistically valid sets 
of courses recommended. 


Keywords 
Recommender Systems, Course Recommendation, Confor- 
mal Prediction 


1. INTRODUCTION 


Recommender systems are systems capable of predicting the 
preferences of users over sets of items [1]. They can be found 
almost everywhere in the digital space, shaping the choices 
we make, the products we buy, the books we read, or the 
movies we watch. The range of applications of recommender 
systems has been broadened recently to the education do- 
main, especially in higher education [5]. There are sys- 
tems reported that provide recommendations for academic 
choices, learning activities, learning resources, and learning 
collaborations [14]. 


Among the recommender systems for academic choices, there 
exists a particular interest in systems that recommend courses 
[3]. There is a wide range of such systems that differ in the 
underlying recommendation mechanism, accuracy, type of 
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recommendations (courses, course sequences, course concen- 
trations), and type of representation. It has been recently 
recognized that course recommender systems need to be safe 
[11]; ie., course recommendations need to be provided with 
confidence information that will help a student to make a 
better course selection. There exist different approaches 
to delivering such confidence information from course pref- 
erence ranks estimated by the underlying recommendation 
mechanisms [3, 6, 10, 12] to separate warning modules [11]. 
The characteristic feature of these approaches is that they 
are heuristic, and thus they do not provide any theoretical 
guarantees for the quality of course recommendation. 


In this paper, we argue that course recommendations need 
to be supported with statistical confidence. This confidence 
will allow computing a statistically valid set of recommended 
courses that contains courses a student is likely to take with 
a probability of at least 1 — € for a user-specified signifi- 
cance level e. To achieve this, we employ the well-known 
conformal-prediction framework [4, 15, 16]. We design a 
generic algorithm for conformal course recommendation ca- 
pable of computing statistically valid sets of courses for stu- 
dents. The algorithm is used for implementing two con- 
formal course recommender systems that employ a content- 


based recommendation mechanism. The first system is instance- 


based, and the second system is an exemplar-based system 
[13]. 


The conformal course recommender systems have been im- 
plemented for academic advising of University College Maas- 
tricht, a Liberal Arts Bachelor study with an open curricu- 
lum. In this study, students personalize their program by 
selecting courses that align with their academic and per- 
sonal interests. In total, students choose around 40 out 
of the 160 possible educational modules; i.e., they create 
a program by selecting one path out Cy) possible. Our rec- 
ommender systems are tested to facilitate this process. The 
initial experimental results show that the systems accurately 
recommend courses while providing statistically valid sets of 
courses recommended. 


The rest of the paper is organized as follows. The related 
work is provided in Section 2. Section 3 formalizes the task 
of course recommendation. The course and student topic 
models used for course recommendation are briefly described 
in Section 4. Section 5 introduces the generic algorithm 
for conformal course recommendation and its instantiations: 
the instance-based and exemplar-based recommender sys- 
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tems. The student-course data is described in Section 6. 
Section 7 provides the experiments and discussion. Finally, 
Section 8 concludes the paper. 


2. RELATED WORK 


Course recommender systems received significant attention 
since the very first publications [12, 18, 17]. Meanwhile, 
these systems have become very diverse. Following the main 
trends in recommender-system research there are different 
types of course recommender systems: content-based sys- 
tems [10, 11], collaborative-filtering systems [3], hybrid sys- 
tems [3, 6], and popularity-based ranking systems [6]. Most 


of these systems are capable of providing (explicitly /implicitly) 


confidence information for course recommendations. How- 
ever, this does not give any guarantee for the quality of 
course recommendation in a statistical sense. 


Confidence-based recommender systems have been proposed 
based on collaborative filtering. The first system is based 
on group recommender systems [8], and the second one is 
based on matrix factorization [7]. Both systems can be di- 
rectly applied for course recommendations, however, under 
assumptions typical for collaborative filtering. For example, 
plenty of data is available; the course order does not matter. 
In this context, we note that we propose a generic algorithm 
for conformal course recommendation that is not tailored to 
the recommendation mechanism: collaborative filtering or 
content-based filtering. The only requirement to apply this 
algorithm is to have a function that estimates the typical- 
ity of a course w.r.t. other courses taken by a student (see 
conformity functions in Section 5). 


3. RECOMMENDATION TASK WITH CON- 
FIDENCE 


Let T be a set of topics t considered in a set C courses c. 
To indicate the degree of presence of topic t in course c € C 
we employ weight w.,. Topic weights we, of course c € C 
represents a topic model of this course. We assume that the 
topic models (i.e. topics’ weights we,t) are provided initially 
for all the courses c € C. We describe our approach to derive 
these models in the next Section. 


The courses c € C' are given for a set S of students s. To 
indicate the degree student s € S masters topic t we employ 
weight ws,;. Topic weights ws: of student s € S represents 
a topic model of the student w.r.t. courses c € C. Thus, 
they are computed w.r.t. set C',; of courses student s has 
taken; i.e. for any topic t € T we have: 


a 1 
«= eg (1) 
If we assume a specific ordering of the topics t € T, then: 


e the topics’ weights w-,: for course c form a topic-model 
vector We for c, and 


e the topics’ weights w, for student s form a topic- 
model vector ws for s. 


The topic-model vectors we and ws “live” in the same space 
W. Due to the number |T| of all the topics, we employ the 


cosine similarity over W. It can be used to compute simi- 
larity for any two topic-model vectors that represent courses 
and students. 


The topic-model vectors we of all the courses c € C form 
the course data set Wo defined as {we}cec. Analogously, 
the topic-model vector wz; of all the students t € T form the 
student data set Wg defined as {wi}zer. In this context, 
we introduce the recommendation task considered in this 
paper. Given a course data set Wc, a student data set Ws, 
and a student s € S with topic-model vector ws € Ws, the 
task is to compute a recommendation set C§ C C\ Cs that 
contains courses that indeed fit student s with a probability 
at least 1 — € for a predefined significance level e € [0, 1]. 


4. COURSE AND STUDENT TOPIC MOD- 


ELLING 


We employed the topic-modeling approach proposed in [11]. 
The set T of topics t was identified from the course descrip- 
tions using the Latent Dirichlet Allocation (LDA) generative 
model [2]. Each topic t € T is given by a probability distri- 
bution over the vocabulary derived from all the descriptions. 
Thus, each course c € C' is represented by topics t, which 
words are present in the description of that course. Student 
topic models are derived based on the topics courses using 
formula (1). 


5. CONFORMAL COURSE RECOMMENDA- 
TION 


This section introduces a conformal course recommendation. 
First, we present a conformal test for course inclusion and 
a generic algorithm for conformal course recommender sys- 
tems. Then we provide two instantiations of this algorithm. 


5.1 Generic Conformal Course Recommender 
Consider a particular student s € S with her set C; of 
courses. We assume that student s is represented by a proba- 
bility distribution P;; i.e. P; has generated the course set C's 
for s. Thus, to decide whether to recommend a new course 
c ¢ C, for student s, we perform a statistical test of the null 
hypothesis that the set Cs U {c} is generated by the student 
distribution P, under the exchangeability assumption [15] *. 


We implement the statistical test according to the conformal- 
prediction framework [15]. It makes use of course conformity 
scores. The conformity score a. of a course c is defined as a 
score that indicates how typical c in set C;U{c}. The confor- 
mity score @. is computed by a course conformity function 
A. The latter is a mapping from 2° x C’ to RU{+00}; ice. it 
returns for any course set C's and any course c a score a@- that 
indicates how typical is course c for the courses in C’; U {c}. 
Depending on the implementation of the conformity func- 
tion for the course and student topic models, we can have 
recommender systems based on content /collaborative filter- 
ing (See the next section). 


The conformity score a, of a new course c is used as a test 
statistic for the null hypothesis that the set Cs U {c} is 
generated by the student distribution P; according to the 


‘We note that the exchangeability assumption is weaker 
than the well-know i.i.d. assumption. 
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exchangeability assumption. The p-value p. for the null 
hypothesis, is calculated as the fraction of the courses in 
C; U {c} associated with conformity scores that are equal 
to or smaller than a.. The larger the value of p., the more 
likely it is to observe the value of a. under the null hypoth- 
esis, and the more confidence we have in course c. If we 
set a course significance level €. (probability of the error) in 
range of [0,1], then the statistical test will accept the null 
hypothesis if p. > €- and course c will be recommended. 


If we perform the statistical test from above for student 
s € S over all n courses from set C'\ Cs ( that s has not 
taken), then we can compute a recommended set of courses. 
We summarize this process in Algorithm 1 given below. It 
presents a generic conformal course recommender algorithm. 
Given a course significance level e. € [0,1] and set Cs of 
m courses taken by student s, the algorithm computes set 
Cs © C\ Cs of recommended courses for student s on the 
chosen significance level «. To decide whether to include 
course c;, € C'\ Cs in set C§ the algorithm computes the 
conformity score Qe; for each course c; € C;U{c:} using the 
nonconformity function A (step 4). The conformity scores 
Me, are used for computing the p-value p-, of course c; (step 
6). Once pce, has been obtained, course c; is added to the 
set C$ of recommended courses if pe, > €¢ (step 7). This 
process is repeated for every course c; € C\ Cs. 


We note that the generic conformal course recommender al- 
gorithm computes valid recommendation sets C§ for a sig- 
nificance level € that usually is bigger than the course sig- 
nificance level e- (used in Algorithm 1). To explain this 
phenomena we follow the approach proposed in [9]. W.l1.o.g. 
assume that the set C’ \ Cs is the same for all the students 
s € S. Let ec, be a random error variable for a course cj 
from the n courses in C \ Cs. The variable €c, equals 1 
if course c; does not fit student s; and 0, otherwise. As- 
sume that we set the course significance level €. so that 
p(€cy = 1) < €c, p(Ecg = 1) < €c,..., p(€c, = 1) < €c. This 
implies that the expected number of courses incorrectly rec- 
ommended, €c, + €co +... + €c,; is bounded by ne;; i.e. 
EDD ,cc\c, Ceci] S Nec. If we know number t of courses 
from C\ C, that fit student s in advance, then: 


SEL D> ead < Zee. (2) 


cEC\Cs 


We note that FED e0\c, €c,| is the expected error and 
*éc is a significance level € for which validity of recommen- 
dation sets C'§ can be established. This implies: 


€o = —E (3) 


Thus, to guarantee valid recommendation sets C$ C C\ Cs 
that contains courses that fit students with a probability 
at least 1 — € we need to set the course significance level 
€- according to formula (3) when we initialize the generic 
conformal course recommender algorithm from Algorithm 
1; 


Algorithm 1 Generic Conformal Course Recommender 


Input: Course significance level €., 
Set Cs of m courses taken by student s. 


Output: Set C§ of recommended courses for student s. 


: Set course set C$ equal to 0. 

: for each course c; € C'\ C; do 

for each course c; € C; U {c;} do 
Set conformity score a. i of course c; equal to A(CsU 
{ci}, ¢9). 


end for 


EC SU{ Cy co; Se, 
Set pe, equal to uae oe i Ee 


Add course cj to C$ if pe, > €c. 
end for 
: Output set C’S¢ of recommended courses for student s. 


To establish the validity of sets C$ of recommended courses, 
we adapt the error metric from [9, 19]. Assume that for 
student s € S, we have a test set of courses, and we know 
that within this set, there is a true set C! of courses that 
student s will take. We define the individual error e, for 
student s € S as the proportion of the courses in the true 
set C% that are not recommended, i.e. 


_ ICs\ (CSN C5) 
IC3| 


€s 


In this context, the error e of a conformal course recom- 
mender is defined as the averaged error e over all the stu- 
dents s € S: 


a Dises es . 
| S| 

We note that the individual error e; corresponds to the ex- 

pected error eq EDejec\c. €c,]._ Thus, to show experi- 

mentally that a conformal course recommender is valid for 

any significance level € in [0,1], we have to show that the 

error e is less than or equal to e. 


The validity of a conformal course recommender can be triv- 
ially achieved if the recommender outputs all the possible 
courses from set C’ \ Cs. Thus, we need to estimate the in- 
formational efficiency of the recommender. For this purpose 
we employ the size SR of the recommended set C$ of courses 
averaged over all the students: 


dses |Cs| 
SR= Ss 
|5| 


5.2 Content-based Conformal Course Recom- 


mender Systems 
The generic conformal course recommender algorithm can 
be instantiated if we specify the course conformity function 
A. This function can be done using different recommender 
mechanisms, e.g., collaborative filtering or content-based fil- 
tering. In this paper, we assume the existence of topic model 
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vectors of courses and students that fit the content-based 
filtering scenario (see Section 3). That is why we propose 
conformity functions for two content-based conformal course 
recommender systems specified below. 


The first system is an instance-based conformal course rec- 
ommender system (ICCRS). Any student s € S' is repre- 
sented by a set of topic-model vectors (instances) we of the 
courses c € C’; she has taken. In this context the course con- 
formity function A outputs for any course c € C' and course 
set C's of student s € S an averaged similarity of c with 
courses in C; i.e. OATH ic EC,\ {c} CO8(We, We’) Where cos 
is the cosine similarity. 


The second system is an exemplar-based conformal course 
recommender system (ECCRS). It employs topic-model vec- 
tor ws (exemplar) of student s € S computed using formula 
(1). In this context the course conformity function A out- 
puts for any course c € C and course set C's of student s € S 
a value equal to cos(we, ws), where topic-model vector ws of 
student s € S is based on the courses in C’; \ {c} and cos is 
the cosine similarity. 


The computational complexity of ECCRS is higher than 
that of ICCRS since, for any student, we need to recom- 
pute her topic-model vectors ws by excluding courses one 
by one. However, ECCRS has better explanation capabili- 
ties. The topic-model vector w, of student s represents the 
current levels of topic mastering, and the topic-model vector 
We, of course, c represents the topics covered in the course. 
Thus, the cosine match can explain why the course has been 
selected /rejected. 


6. STUDENT-COURSE DATA 

ICCRS and ECCRS have been implemented as course rec- 
ommender systems for University College Maastricht. The 
college has provided course enrollment data from 2008 to 
2017. This data includes course and student identifiers, 
grades for each course, details regarding course assessment, 
ECT credits, and course descriptions. The course descrip- 
tions facilitate the construction of topic values for both the 
student model and the course model. The calculation of 
topic values is with LDA, and an optimal number of top- 
ics is determined through maximum likelihood estimation. 
This optimization results in sixty-five topic areas represent- 
ing the course catalog [11]. We remove modules without 
descriptions from consideration. In total, 143 courses and 
2422 students enrolled in at least one course remain. 


The rates of course enrollments vary widely between each 
course. Registration in the majority of courses offered oc- 
curs only a few times over the entire period, see in Figure 
1. The modules provided are updated each year, reflecting 
the changes to the course catalog via dropping courses and 
course code changes. Several introductory courses, required 
courses, and projects make up a significant portion of all 
enrollments. Most students at UCM need eighteen periods 
to complete their education. Nevertheless, some students 
enroll in over twenty periods. See Figure 2. Each recom- 
mender system focuses on a subset of twelve periods repre- 
senting two years at UCM. The subset is refined further by 
selecting only students starting in the fall intake semester. 
These restrictions increase the standardization of students 


for our systems, and balance for the diversity of enrollment 
patterns present in an open-course curriculum. Our recom- 
mender systems use the remaining 1018 students that fall 
within these boundaries. 
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Figure 1: Total Number of Course Enrollments 


3000 
2500 


2000 


iin. 
oO 


123 4 5 6 7 8 9 1011 12 13 14 15 16 17 18 19 20 21 22 23 24 25 


Periods Students are Present 


Number of Students 
bo 
oo. 
So 98 
So. 8 


w 
fo} 
o 


Figure 2: Total Number of Student Enrollments per Period 


University College Maastricht offers a project-based cur- 
riculum. Two of the six periods each year are for stu- 
dent projects (periods three and six). The choice of courses 
within project periods is restricted. See Figure 3. Exclud- 
ing these project periods, the average courses offered each 
period is 31 with a maximum of 44 courses. Figure 3 shows 
the variation in course offerings throughout the UCM data 
available. This variation is taken into account by our sys- 
tems, and we omit these project periods from calibration. 
Course recommendations include only those courses offered 
in the target period. Therefore, recommending the maxi- 
mum number of courses for a period results in a 0% error. 


— — Average # Courses (Excluding Project Periods 3 and 6) 


@ Unique Courses in Period 


Unique Courses Offered 


(Year, Period) for All Academic Periods 2008-2017 


Figure 3: Courses Offered in Each Academic Period 
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This section presents experiments of ICCRS and ECCRS 
on the student-course data provided by University College 
Maastricht (see Section 6). First, an experimental setup is 
given, followed by results and discussion. 


7.1 Setup 

We validate ICCRS and ECCRS on the student-course data 
in the order of study periods. Assume that we have M 
number of periods P,,..., Pn in which a student studies 
towards her degree (for our data M is equal to 18). We 
denote by Cs(Pm) CG Cs the set of courses that student s 
has taken in period P,, form < M. Given new period Pm+1 
together with the set U™_4iCs (Pm) of courses student s has 
taken before that period, we test our recommender systems 
by checking whether the recommended sets C’§ of courses for 
Pm-+1 includes the courses Cs(Pm+1) that student s indeed 
has taken in Pm41. 


IDI FP ID TP IPT» 
——— 


Courses Taken in 
Previous Periods 


Actual Courses 
Taken in Period m+1 


Figure 4: Prediction Process for Period Pm+1 


We validate ICCRS and ECCRS using data from students 
in their second year of study. This choice is due to the fact 
the number of p-values possible is related to the number of 
courses a student has taken (see line 6 of the generic con- 
formal course recommender in Algorithm 1). For example, 
a student with only three courses taken in period P, of year 
one can only have p values from the set {4, 4, 3 1} for any 
new course in period P93. 


For the validation process, we estimate the average error of 
e, and the average size of SR of the recommended sets C§ 
of courses. We then use these statistics to study ICCRS 
and ECCRS as conformal predictors and as recommender 
systems. 


In the first study, when we investigate ICCRS and ECCRS 
as conformal predictors, we are interested in establishing the 
validity and informational efficiency of the systems (check 
Sub-section 5.1). In the second study, when we investigate 
ICCRS and ECCRS as recommender systems, we are inter- 
ested in estimating the error of the systems over the periods 
when we employ the recommended sets C's on a given course 
significance level «. In our experiments, we use course sig- 
nificance levels € of 0.05 and 0.1. 


7.2 Results and Discussion 

Figures 6 and 7 present the error plots and size plots of the 
recommended sets C§ of ICCRS and ECCRS, respectively, 
for course significance level €. 7. The error curves are very 
close to the diagonal (0,0) — (1,1), which means the error 
is close to the course significance level «-. For ICCRS, the 


2We use the course significance level €. instead of the signif- 
icance level « for the predicted course sets since the range of 
e is very restricted according to formula (3); e.g. [0,0.025] 
for 40 possible courses in a study period. 


error is bounded mainly from above. For ECCRS, the error 
is bounded mainly from below. This bounding indicates that 
the systems are valid given sufficient information, especially 
ICCRS, which is conservatively valid [15]. 


The conservative validity of ICCRS explains why the aver- 
aged size SR of the recommended sets Cs° is higher than 
that of ECCRS. Thus, we may conclude that the informa- 
tional efficiency of ECCRS is better in our experiments. 
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Figure 5: Period errors of ICCRS and ECCRS on course 
significance level €, of 0.05 and 0.1 


Figure 5 presents the error of ICCRS and ECCRS when 
recommended sets C'S on course significance levels €. of 0.05 
and 0.1 are used. The systems are applied over periods of 
P,, P2, P3,P4,Ps, and Ps of the second year of the UCM 
students. The results show that: 


e ICCRS and ECCRS produce accurate recommended 
sets of courses with an acceptable error. 


e ICCRS is more accurate than ECCRS. This difference 
can be explained by the fact that ICCRS is more con- 
servatively valid. 


e the course significance level €, plays a substantial role: 
for 0.05, the error of recommended sets C§ is much 
lower. However, this comes with a price: the size of 
the recommended sets is bigger when epsilon is lower. 


8. CONCLUSION 


This paper shows that safe course selection can be obtained 
if recommendations are supported with statistical confidence. 
The statistical confidence can be used for computing a sta- 
tistically valid set of recommended courses that contains 
courses a student is likely to take with a probability of at 
least 1 — € for a user-defined significance level e. 
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Figure 6: ICCRS - Calibration and Course Set Sizes 


We have developed a generic conformal course recommender 
algorithm that outputs recommendations supported by sta- 
tistical confidence. The algorithm has been instantiated in 
the form of two confidence-based course recommendation 
systems. The systems are essentially content-based: the 
first is an instance-based recommender system with rela- 
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ECCRS Year 2 Period: 4 
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Figure 7: ECCRS - Calibration and Course Set Sizes 


tively high accuracy. The second system is an exemplar- 
based system with a lower accuracy but with better ex- 
planatory capabilities. The experiments showed that both 
systems accurately suggest courses to students while provid- 
ing statistically valid sets of courses recommended. 
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